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1. INTRODUe'UON - > ^ • . * 

1.1 Overview . * ■ . ^ '. 

This report describes an wtperimental systen for drawing simple 
pictures on computer graph^ds' terainal'i/^i* natural. language input. Tne ; 
systea is capable of drawing' lines, points, and circles on coEaand from the ' 
-user, as well as answering questions about system capabilities and objects ' 
on the screen. Erasures are also Remitted. , Language input' can be em- 
belltshed with touches to convey positional information., 

rne system wg^s designed and implemented by the authors during Summer 1976 
was written in ifsP 1.6, runs in aJjoUt 40K words on a i>ECSysten-lb ' ' • 
computer, and displays pictures on an ag60 Plasma Paij/l. 

The sys-tem was implemented tc test out Ideas on syst^ organization, 
-to establish the viability of combining language and grafjhics. and^to 
experiment with appropriate A. I.- techniques. ' ' * . 

1.2 Example of dial'ogue . ■ - ' ' ' 

*? PLEASE DP.AW A VERTiCAL 2 LSCH LINfe ^ <T>-. '• . 

*? ?LT A POINT CALLED FP^D HERE <T>. ' ' • v - ' • 

OK , •■ ' ' , ' ' - 

*? MAKE A CIRCLE 'rflTH A TEK CM DIAMETER AT FRED." 

OK • - ' • ' . 

*? CON-NECT FRED AHD (100,150) 

OK ' . ■ • 

*.? CALL THE CIRCLE BALL ' . » 

OK , -• ■ ■ ' • , ■ 

*■? ERASE THE LINE FROK FRED 
OK - 

*? ERASE FRED ' , 
OK * ■ • ^ . ' . • 

*? CALL THE 2 INCH LINE BXT.. 

OK ■ • ' . ' ■ . . 

*? DRAW A CIRCLE ' * ' • ■ ■ ■ 

Ok ^ ^' ' - ' 

*1 NAME THE CIRCLE FACE ' T ' . 

OK ^ . ^ 

*J WHAT DID YOU- DRAW HERE <T^ • • . • 

a LItJE called BAT , ' . ' , ' , 

*? WHAT CAN YOU DRAW? • ' • ' . • • . 

LINES, POINTS, and CIRCLES ' ' '"• . • - , 

*? IS THERE A CIRCLE CALLED FACE? - ' ' 

yes " • ^ ^ . . ' . • 

*t Zm TOU DRAW SQUA82S? ' • • . \- ' 

f ■ . . ' / 

The system proB^t Is and the systeer te^p<mS^_U oil the foXlUing' \. 

line. The symbols "<T>" indicate a touch on -d^e spreen.' TMs extract"" ■ " 
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• " • ■ ' •■ ' • ■ - . - 

'shows some of the variety' of sentential forms available in the system. Not 
-shown, are the results pf the commands on /the. scre^, > ^ 

1.3 Motivation ^ * ' • *.' 

The-.research'^being dqp^ by our group' promiseV to begone more step , 
towards hjie goa^l of naturaj. interaction between maa.Vnd computer. Our 
work is based om fhe belief: tha< use of more lhag one mode of cimmudica- 
tiort is required to achieve, that goal.^ We Jiave chosen 'to .investigate the 
_combinat.ion of natjl?al language and graphic^ 7^ system using this combi- 
nation allows the use of "linguistic, g^kphical, ox inixad forms for botff ' 
^ input, and output of information.' Cafeful development of this idel would 
/ provide practical systems wit^i- a high degree of habitability. 
. ^ .Many artists and designer^ who are (and wish to remain) ifelve about - 
programming wiJJ be able to inter'act productively ^ith a natural language 
•graphics system and such a system would" allo<r a much wider" grOup" of people 
to use compters. Special subsets of. both language and pict™s/&n be * 
, developed for various uses.' We are adv^ocatin^a form of ,atW^ language 
programming, but with an- addidcmar (.graphical) mode of ccmmi^ication. 
We feel that the feedback* provided by the- graphics will assist the user 
m detecting, and^hen interactively correcting, errors due cp vagueness 
^ ambiguity. In addition, the user is able to sSlect,the appropriate 
method for inVut of informa-tlon and, for example, could provide a iSugh 
drawing of an object with a description of additional details. These ' ' 
techniques would be viable ii> a variety of applicatioos area^, such as 
Animation, Jtrchitecture, Engfaeerlng, and Education. . 

Natural Unguage Graphics tlJLG) provides a framework for rfesearH in 
s/veral areas, of computer science. It supplies a domain jfor^ the study of 
linguistic phenon^ and language understanding systenTdesi^. MemirtTf* «,d - 
inference will ^ay a large role lb an NLG'system and- ^equently 

knowledge repre^ntation and manipulation is important. Such systems 
would also lead to new idea^ about Computer Grpahics. helping .to liberate 
if from its. present algorithmic approach. ' >fen-mach^e interaction studies 
and AI systems organizatioii, wi^l al«o need to be ptnrsued. • . " 

■ There .has been no systematic -sttKiy of the combination of language 
^nd graphics, although .thei:e have fikn a few sykeas'wlt'h this t^^binafion 
fICirscbr Coles, Sim^n^rSadler and Wino^adJ and 80»e recent work on " s 
graphical information [Sondh^imer, 'Pafiher. Agin, Hevatia HtoskyJ. 



1.4 Report Outline ' ^ ^ - 

>The next section, 2, .is concerned with the overall design and operation 
of the syst'em. Sections 3- and 5 describe th*- language processing and know- 
ledge components respectively. ,As these are the longest and most impojrtant 
components they are both described" in *sbme detail. The" language " " ' • 
generation component is prTsentgd in -Section 4, -while graphical output is 
described in Section, 6. The final sectiotx is a nummary and .conclusion. ' ' 

' » % ' ' ' • 

-2.- - OVERALL SYSTEM DESfgK ' ■ " ' 

■* ' ^ i 

2.1 General Description ^\ . , 

NLG is composed 'of nine independent modules. Coi^nicatidb among 
iKJduLes is achieved by a message-passing s'cheme in which "any module may- 
invoke another'by sending an appropriate message. As illustrated in Figilre 
SYS-1, all messages are handed to the Executive module (EXEC) which, 
ultinately relays the message to its destination. Thus ^he system i^ 
organize^heterarchically, although a spec ^ galling sequence has been 
induced (on the model which limits the interaction of modules in -actual 
operation. ' . 
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• ■ ^ J . . . ' • • . ■ , ■ 

Each of tXe remaining eight modules iias specific duties in the pperation 
of the system^ Th4 Initialization module (INIT) contain^ startup infor- 
* , mation for NLG. If the system detects internal inconsistencies. th*e System 

Error module (ERR) is called to interact with. the programmer in debugging,, • 
■j • Modules for graphical (touch) inpufc- (GI) and typed text^np^ut (IN) 

provide prepitocessed- user input for languag'e analysis. » The Langi^e 
Aiialysis module (U^j ■ parsers tbe ^npQt, using an ATN grammar, into a case- 
based semantic structure. The Knowledge Base njpdule (KB) develops a 
semntic network from the input and creates 'instructions' on how to update 
the display, screen. The Graphics Output module (GR) uses\hese instruc-' \ 
■ tions to run a graphics program. Responses to th^ user are pi^odu'ced by 
- t^e^ Language Generation module (GEN) using a generative ATN grammar. 
NJ,G is written in LISP 1.6, except^ for the graphics primitive 
routines which are coded in I^CRO-10. As a debugging feature, "the evaluation 

: ■ of a LISP expression *i5 ^permit ted whenever the system prompts the user for 

input by preceding the expression with a dollar* sign character ($K 
^ Each module or collection^of data used in building the system is 

• ^ .. contained on a separate file. (A* complete breakdown of the size of each 
module ippears in Appendix A.) In ^addition, an assembly of utility ^ 
. functions and functlpns useful in more than c^e madule is kept on a 
single file. .The original motivation for doing' this was to *facilftate the 
dynamic swapping of modules to maximize the use of' memory. Although tlie . 
me^anisms for this were implemented, processing time' increased prohibitively. - 
HJjii^er, the complete separation of modules in this "fashion eliminated 
many of th6 problems typically encountered in developing large systans. • 

\^ 2.2 System Construction and Starfup ■ ■ " ' 

^J^^yy^^^^^ ^'tists in LISP 3nd MACRO^IO form.' For ease of construction, • 
a special control file ig submitted to the monitor to build the system and ^ 
; create a core-image "SAVE" file. TBe contend of ttie control file 

follows: " *. , . ' - - X 
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.CCMB LAftirHJLSPF4,fPLASUB 

.R LISP 35;/A " * ' , 

1200 12400 ' • / 

*(SETQ *TRACEOFF* T) ' r 
*(PUTSYM FLONUM FIXNUM) • ' 
*(LOAEL NIL) » . 

*MRiaH,ILSPF4,PLASUB f 
*(DSKIN (NLGLP2 . LSP)) ' 
* (DECIMAL) 

MGC) • - . 



^ ' /■.SAV PLASMA. SAV - \ ' 3.0 

J ir " ' • 'DgL PLASMA.HGH \ • - 



/ 



' where '$ re^esents an altmode character. • . / 

First, the three MACRO-io files are compiled: the file LARITH 
estatli^h^s th| interface between' LISP and' the ■ FORTRAN library arithmetic 
routines (i.e., SIN, COS, etc.); the file ILSPF4 builds the interface 
from LISP to the graphics primitives package; and the file PLASUB contains. 
. the plas^ panel subroutines. LJSP then runs in 38K with .the full word 
^^pace set at 1200 words-" (octal) and the frfee .storage. sjace set at! 12400 
words (o6tal). These values were established, mostly through eijcperimen- 
tation, but see.fhe LISP reference manual [Quam and D>ffie] for fuftlfer 
details. Once. in LISP, the iCB trace mechanism, used for debugging, 
nLst be switched off by setting the flag *TRACEOFF'^. The two^int^rnal 
LISP routines FLONUM and FIXNUM, which deal- with converting fixed .and 
'floating numbers, must be put on the DgT symbol table po be available for ' 
the interfaces. Next , 'the' relocatable files compiled earlier ar% loaded 
into expanded core. LISP makes use of the system loader which itself 
is loaded on call and unloaded when finished.* At this point, the total 
' core allocation for NLG may reach 43K. • The system is loaded next. .The 
file NL6LP2.LSP contains the individual file' names a^ong with their 
Location on disk. Ustly, the system is'placed in decimal mode and' garbage' 
collected. The complete system is now .saved in the core-imag-e file 
"PLASMA" and the task is completed. The sketchy description given above 
may be clarified singificantly be referring to the LISP manual's 
sections oij adjusting system parameters and loading binary files. 

Once a core image of NLG is available, a user types ^ ■ . . 
.RU PLASMA - , . 

to gain access to it. At the prompt ^ a valid message to INIT may be sent 
via ^JJEC; however, for convenience, one may type 

(INIT) . ■ ' 

to 'establish contact with NLG! 
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2.3 The E?cecutive Moduli m ' ' - * , 

The Executive module; (EXEC) receives and sends all messages in the ^ 
system, A validity t^st is performed on messages according to the following 
syntax . ( ^ * . 



The rfotation used in expressing syntax in this report is a modified 
•BNF where. the -symbol "or" sHbws altemat^ives and subscripts following 
non-terminals show the minimum number of occurtences. 



<message> 

<fr9m> • 

<mark> 

<args> 



- (•<to> <fr9m> .f.mark>-.<arg^s> ) 
= <mo<}u]ie 9ame> * ' » ~ • ; ^ , 

- <njodule name> » 

- J.Q or ifR ' ■ , ' 
' ( <ari7„ ) , ■ ' r ^ ^' 

^.The.-module.names used in ^he <to> and <from> components must correspond ' 
to the names ^ven in Figure SYS-1. - ^4 list o'f argumVnts is not checked 
by EXEC, Mnce each module generally protectsi ifsei^ f ronr tad data. * - 
'M,ssag^s With .a query n>ark'(?Q) are sent as module c^Us while tho^e 
with-a response mark (M) are returned as a'value to.the cal-ler. AI30 ' 
EXEC may be given the message by calling or returning. This allows" the ' 
conveyance of messages in fouk- ways'. cofresponc{ing .tb t^e manipulation of 
the cal^ng. level in-LISg^and the calling level of NLG, ThV four / ' ^' 
possibilities are-: •. # "• • . 



(1.) calling EXEC. with a SQ message; ' • 

(ii) calling EXEC with a #R message; 

(iii) returning a (fQ messag'e to EXEC;' and - 

- . (iv)_ retufning a .^iR message to EXEC. / ' '• 

The first Combination results in a conventional module call while the last 
results in a canventional .rettfrn. The second oossibility is never used 
. in NLG. The third one ahows module" invocat:io?in a manner s'lmilar'to ' 
co-routining, 'LA and KB U>^ionally communicate in this fashion." 

EXEC'S final du^y is to automaticaily produce d ^ce of all system " 
,messag.s on the file MESS.LPT. including statistics on time and spac. . " 
Utilization. - * • ' " 

c • ; . ' • . • • ' . ^ * 

' 2,^ The. Error Module • ' ' . * . 

The System Error module'(ERR) exists solely as a debugging feature. 
Ir the message vali^.itf test conducted iS^ E)&C fails, then ERR is' called 
It displays the. bad message and inteVac(tg wich The user to create a^new • 
one.^ Of course, the dollar sign ($) feature which allows lIsP evaluation 
is supported allowing a knowledgeable perspn to completely investigate 
the pr-^lem. ERR may b^e co;i,sidered an Extension to EXEC, but has been - , ^ 
rarely used. ^ * 

' . - , \ ^ . 

2.5 The Initialization Module « % '''^ ' 

The system is activated with i user message to the 'initiaiization 
-dule (INIT)-, Here; all global syst.tn-t.ara&eters a.e initialised, output 
channels are established, and an Initial .c^e^n -erase is perfo'rmed. The - 
screen's cursor, position is set to zero and the distance betwee. lines ' 

. • .12 
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of text cm-CHe-scxeen_i8 se£\o 15' strfeen units. These parameters are 
utilized in^-eposit inning the cursor "afrer| drawing. An introdyptory - 

systrem response is then made via GEN. , "^^ - • 

Next;, a message is sent to activate' LA for input processing. It ~~~~ 
■must be 'determined f^ojn the eventMal reponse to this 'message whether 
to srfop the. system 'or to r6-activate LK. .For t-ltc former , an exit ' ' 
responpfi is made and NLG halfs,' otherwise GEN is directed to Respond 
"OK" and-LA fs called again. . - 

2.6 TypicaX Control Flow ' ' , ' • 

^though "is base^^o> a heterarchical model, hierarchickl re'la^l/on 
ships exist among modules viewed in oi/eratio(i! The followlt 
illustraties typical x:ontrol flow in the ssystem^; 




fNIT- 




GEN, 



Gl 



Vfhen -^IT is activated by the user, 'it knows, to. expect- Input . ^Thus, "LA^. 
is called. - From there, IN is activated to process the raw input ,. with 
GI being called to han;He graphical input ! During LA''s processing, KB ^ 
may be consulted for Storfed or acquired knowledge. f»oiv completion, LA 



Of course, EXEC'.controls the passing of messages, but the diagram would 
become unnecessarily complicated if EXEC were include'd evetywhere. The 
• remainder of^hlS paper will contain, no 'mentioii of EXEC unless essential 
to tbe discusBipnj. ' • , ' 



The fopner happens' for 



can .either- •pi.ss control to p or retusn.^tfe INIT. ' The 
"-^'c-opnands and th^.iatter ^r question-answering, f When KB gets control, 
^ 'it concludes' the processing by buUding memory fCructutes and cftrec&iijfe 
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GR to draw. . KB' then returns to' INIH. 
LA, -or KB. ^ 



3.^ THE LANGUAGE ANALYSIS. m\mp ^^ 
''■^.1 Function an^d Overall 'Design. 



GEN may^be activaje'd by either INIT, 



The Language Analysis izfodple (LA) processes all input to the system. 
Including both typed text aYid touches to' .the screen. Figure LA-1 #how^ ' ' 
the logical structure- of LA. * ' f ' • 




ERIC 



^ ^ ' figurb'la-i ^ 

/ • . la lOrgansization » " ' " 

An Augmented Transition Network (ATN^ parsM^^ods, '1973; series as 
th| primary component of the module, and aicall of -LA may optionally 
sp^i^ an ATN state name or configuration. WhUe parsing, the gi:ananar 
permits .queries to the Question-Answering (QA) mechanism ^thli the ' 
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Knowledge Base (KB), ^via the functioit QACALL. to ratrie.ve other 'stored 
knowledge as required. .The parse results in a case-Jike- structure called 
/ a prototype, which contain.s a representation of the original ^nput (see 

Section 3.4). If the in^U* posed _a question, the answer may then be ^' 
• ■■ retrieved frota QA via thfe function, QASK. For cpcHnands. the f.rototype • 
is passed as the argument iitya message to the" Memory and Graphics (MG) 
^mechanism^within KB." If the in^ut commanded the system to stop, then a . 
•return message is conveyed to INIT which stops jhe systenf. 

3.2 Tne Two Input Modules / 

Before parsing can begin, the input must be scanned and assembled int 
list form. The Input module Un) is designed 'for this purpose. If'lN 
should detect a touch control character while scanning, then the Graphics 
•Iriput module (GI) is activated to dnterpret a touch. Touches become 
screen coordinate T,airs (a L-fSP doe'ted pair of numbers) in jtfie input list. 
Encountering, a 'carriage-return causes IN to return the input list to LA' 
for» parsing. 

^3.3 Parsing, 



"3.3.1 The Augmented Transition Network Model 

The ATN parser nsed tn this njodule is modeled after the one used 

.in Uxr^AR system [Woods^ 1973; Woods, et al.. 1972].^ A finite state 
net^rk'of states and arc?^ is endued with recursive capabilities allowing 
a sub-network, trave^Jsai' to be performed in the course of following one • 
arc. This recursive transition networ)c. is >rther aiigmented by the 
capacity to manipulate "registers" and perform function caAs daring 
traversals. Such registers can contain flags, .words' from th'e Input- 
string, ,partial parse trees, case structures, or apy expression permitted 
by the host language. This allows the yarrious constituents to be built, 
tested, altered, jand steered into positiou within the representation - 
chosen. The behavior ^of the parser is determined 'by the grammar and lexic 
^ provided. The syntax of grammar specification in an ATN appears in 

Appendix B. The lexicon consists of a list of feature-v^lue 'paips for 
• each word sense. ' ' ^. 

3.3.2 Lexical Entries 

In NLG, new features were acjded 'to .lexical entries .throughout^ 
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the project as^,^^^becomr justifiable af^er discovering additional 
distinctions'. t^ bO»ede wfii^fe ^^ocessing' ' Four features are considered 
basic, to- the /parsing -scheme :'^{ie ^tactic category 6f the word sense 
(CTGY),'^th'e foot fofm of the entry (ROOT):f; the number Category (SIHG oc * 
PLUR) for .nounff-^CKUM) ;. and the transitivity of ye'rbs (TRANS), A case 

^^^''"''W^^^?-'"^^ ^^^-^ "^^'^ Co^Siiggest wliich case ot^ cases the entry 
might m. .Fo*i the position cas^^, an indicator W) ^showed the tjfpe of : 
positibfl*^eing, described b^- taking the values BNDPOINT\or LOCATOR. For 
verbs'like "connetf which 'literally n«ans "draw line" Lhe object to be, 
drawn (i,e^ a .straight -line) 1^ specified, under the fegVure OBJ. Words 
such as "circumference-* or "diametepwhich apply only t\ circles have ' 
the feature ZOBJEC^ with*a value of (CIRCLE) £o indicate this dependence,. 
Similarly, Tdegrees" applies ^as a size measurement to angles and is 
indicated by a hzEOF fealAire with .the proper yalue. Adjectival relation- 
ships are utilii^^aJnyisambiguatidn to limit ' tfh^ number of possible 
word senses to con^r. For exan^de, a line 'can be STRAKHT or CURVED 
4 and the feature ADJ Is used to Specify this fact. Determiners can be 
definite or indefinite indicated by a DEF feature with either a T or 
NIL value, La^stly^ abbreviations afe distinguished by t)ie feature' 
ABBREV. me input vocabulairy appears in Appendix C while 'a partial lexic4 



is; given in Appendix"^, 

3. 3. .3 "tiRe^ Grammar * ^ 

The grammar for parsing can be viewed as five- distinct' networks: 
sentence, toucfl. noun l)hrase,^^prfeposi^lonal phrase,*and- question. The ' 
five 'network diagrams appear in Appendices E,* F, G. H, and I. 

The sentence level .network conCcols the construction of imperative' 
prototypes, ^The initial state^(S*) contains.arcs whicl) decide whether a 
€omi&and ou a question form Is pnesenC, . Questions typically begin with • 
auxiliaries or question words of various types and these are^detected by 
.the function QSTART. called as the test portion bf the' arc wlftch looks for 
questions. Notice thafjthe grammar permits prepositional phrases or 
totiches to the scffiHft'at^the beginning of a sente^ice. These are 
remembered using the. HOLD mechanism until ^helx proper place can be found 
later in the string, ' Jhe. imperative form is pkrsed froo\h'e state S*IMP; 
If the system is beitig*cotamanded to s.top. the^-.Slie arc to state S*STOP 
•b^ldsjhe proper message' to be convened t<? J^f and patslng Ito^s. 
Tii^ arc to VPAm^' is cruc_ia^ the develoi^^t of the Inlperatl^e 



prototype. It finds -the verb near the front'.of the's/ntence and estab- 
lishes the prototd^pe. . At/sta.;e VP*HEAD. the post-verbial constituentsUe / 
expected.. In n»st cases, a n<^n phrase will follow, but other possibilities 
ren^^in. For verbs ^ik^ "connect",, a virtual object (ST-LINE) is implie'd. " 
This Will be followed by an 'indication, of what' two .things are to.be connected. 
Once at the state .yP*,^ahy.of a number of constituents can be handled. 
In. pa«ic^ilar, a touch: is treated dire^ly m the 'grammar *in a manner 
Similar t6 aoun phrases and p^r^positional phrases. Whtn the sentence has " " 



been co 



ompl^eiy pSr'sed, the resulting prototype structure is returned as 
the final ^alue... ),''■' ' ' 

The y^uch network allov^ the parsing of a touch to the s'creen which 
has.b^n jlrepro^essed by GI, or equivalently.' a, specif ication in the.input' 
^string^of/a pair of screen coordinates. Either of ^these may'be optionally 
preceded /bj. "here", "there", "this", or "that':. Afper a ^ucjk p''ol5(^ha8 
been parked, the LISP dotted pair of x-y coordinates is returned%4 
result. / An arc has been provided at the accepting state of the tSh' 
network /to recognize multi^e touc^hes to. the Wame pointton tha^^^n. 
This beUe necessary after ^uch a phenomenon was observeTljKperating 
the sy.U. Of course, onl'y o^e pair of ct,o'rdinates is returned in such 

^Tje noun phrase " network describes the Vrocessing of entities used 
in th^sxst^ as noun phrases." In addition to the standard noun-phrase 
consisting of the head npun with nominal modifiers, this network ^so ' 
handles: 

f • ■ • ■ 

, (1), names - e.g., "the point P" or simply "P"; 

(2) ,fehe prepcisition "of"'- e.g., "an angle of 32 degtees" or 
"a line of length two incWs"; ' 

ajt>uc;i usfed as a co^lementizeir e.g, "the lipe <T>"; 
and . '.. ' 



(3) 



(4) caUs yia QAQALL to establish the identity of a name' 
> , , (FINDOBJ) or to change 'numeric scale (CONVERT) . • ' r 

Sufficient flexibility, is provided In the network to process mbst reison^ble 
combinations of the lexical items known to the -system. The-rW is 
referred to the NP* network in Appendix G for a more detailed expositions^ ' 

The prepositipnal phrase network provides an additional flexibility''^ 
in processing, the preposition is found first-. This can J,e eitlfer a 
standard preposition (e.g.,'to, from, at, through", etc.) or one that takes ' 
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a. compound -form (^,g. . "becween) . Next.- either a noun phrase or, a touch 
can be found. Finally, if a compound form"i?expe*cced, "as in 'felwefen 
the point P and here.<T>". then this is parsed.. Note , that by. usisg the * ' 
HOLD mechanism and a^VIH arcv the interpretation given is literally ' 
Vtween the poinj: p'-^nd between Here <T>". 'when" complete the phrase' 
is retuVned and a partial prototype, built from the phrase.' is lifted to 
the calling level. ' • . ', 

rne iinal coinponent of the gramaar is the question network. This . 
has been developed to the point where several types of qv/estioas useful" 
in hXG can be processed correct]*// but remains somewhat incomplete as 
a general question-answering granrnar. Nevertheless, the following questions 
oesult in answers as indicated: ' • ' • 

(1) What can you draw? ^ 

• LINTS, POINTS, and CIRCLES - • " \ 

(2) Can you d^aw circles? ' ■ 

Can you draw a circle-with a 3 inch circumference? ' ' 



(3) 



i^) How many screen^ units is an inclS? 
■6A 

^ (5) ^Is ch€3t$ a point ^namad 

yes ' ^ 

(h) Is t here a circle na^j^ad^red here <T>? * ^ 

no ' * ' w 

/ (7) Is there a straight circle? 

I don ' t know 

•The question net work permits a variety o^f syntactic forms as shown in 
the diagram in Appendix I. but answers cannot be generated for some 'of 

^ihe p'ossible paths; This situation was allowed to develop intentionally, 
since the system is not priiferily concerned with this capability. Kore ' 
work in this area is antiqipated. ^ • * 

The five networks combine to fora a gramaa/ for Kl6 which performs 
well enough to parse most sentences in less than a half second - a major . 
factor in allowi'ng real ti^e response, Append!^ J contains some sample 
sentences with thgir timings. 

3.4 Representation of the Parse - the MG .Interface 

The result of a completed parse ?^ructure called a prototype. 
Appendix 15, contains a descriptiorr of one whi^h shows^the form permitted 
by each of the felots. in the' caae-like structure. The function of each of 
these is 4iscussed briefly below. 



3-4.1 Valfd Case Forns 

• * » 

The %TYPE case indicates what form of input iras detected during 
parsing. Two types a^e distinguished by the_'^tes, namely the Inperativ 
^ (DIP), foim and the- question (Q) for^. Only imperative fonas result in 
calls on K6 in the current iepleoenLtion. Ques'tion fonas -ar^ handled _ 

• completely within LA, _ ' ^ 

Three actions are possible for the ZACTIOK case. These are drawing, 

• erasing,' and naniqe. The verlr^ the imperative senrence is reduced to 
one of these basic forms. ' ' • , 

Only three objects dve recognised ^ the system. Thus, the ZOBJ^ 
case can be filled by either ST-LIKE, CIRCLE", or POINT. 

The ?KAME Q'ase is included to contain the name of the drawn or 
erased object. Fnis nace -is .linlted in" forE to ^ LISP atom and is 
specified by the^s^r In .the, isjjput string. 

. rne^ of ienta/ion of "a line 'dkn be specified in the ZORIENT cas'e. 
J rnis contains a numeric value in degrees tseasured, counter-clockwise " 

from the horizontal. •■ 

Position on the screen can fee^indicated in the 2P0SIT case. Any 
number of points- (usually no oore than two) may be specified. Each point- 
specification consists of a tag for either" EKDPOINT, LOCATOR, or TOUCH, 
and the coordinates of the actual point. EKDPOIKT-is used to indicate 
. the end points of a line. The midpoint of a line, the center of a circle, 
and the location of a point are all indicated by the tag LOCATOR. * A 
TOUCri tag is used whfenev^r the function of the^point is not deternined from 
the input sentence :(e.g«, ""Erase this <T>"). The coordinates can be . ' 
specified in actual screen units as a dotted pair o^/numbers, "or they iay ' 
, be given by rfterring to the no.de identifier for a particular point 

stored in the K3. IJie latter is indicated by th^ tag NODE dotted to the' 
node identifier. 

Finally, the %S.IZt case indicates the length of a straight line or 
the radius of, a circle in screeq units. In 4ddition,^ by including one 
of the tags RADIUS, DIAHETER. or CIRCDHFEREKCE, • the si^e of a circle can 
be given in terms of one of these. ' ~ " 

3.4.2 Creating the Prototype 

The prototjrpe is ^stahlished directlyffron the verb. The- function 
DEFINEPROTO is ]i8ed to creatfe" a ]property list entry for the jjcerb ' . ' 
consisting of a lis^the case; naaes to appear In the finished prototype 
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. . form. At present, 'procofypeg. for draw, erase, and name a're determined 

^° '^^^^ "^y- "i^ist of exactly the same- form, although .this need 

^ , ^ not hecessaiciiy be the. c'a^e. Other verbs, of course, appear in the 

inp.ut.sering, bat these -alte reduced to the canonical verb fo'rm so that 
. ' i^f reference to the prototype i* always done correctly. The function 

• PROTO is us^iUs the AT!,' gransaar.' to retrieve the proper prototype form! ' 
' '\ ^ constituents are discovered and parsed bytl^ ATN graoEsar. 
( fney are slotted into t&e. proper case positions. The function PROTOTYPE 

ehafiges the current prototype to reflect the value fo b& added. Homilly, 
additions are made as they ire^iscoN^red , but the HOLD nethanism/ 
allows soise of these decisions to he delayed until 'a later VIE arcVcaa 
• frSd the proper positioning of tfae constituent.; Tnis is particularly 
<^usefal in handling touches which are found bjt of place and* f<yr moving 

preposicionai phrases to. the post-verblal position. . * ' - 

t 

3.3 rne QA Interface . | 

rne Unguage- .Analyzer utilizes knowledge accumulated in K3 to help 
psi^singf ^nd to answer user questions. , The two functions. QACAli and, 
QASK provide the interface^to QA for gramiBar-produced and user- initiated ' 
que^ions re^spectively^ (see Figure LA-1) * I 

QACJ2.L builds a aessage from the argfeents passed to. it. Toe fi^t " 
argun^nt indicates the nase of the operation required of QA and this- is 
United to either FINDOBJ, CO!?\^T, FINDDRAW, or JHFO. Qther arguments , ■ 
to QACALL are inclGded as part of. the message to QA. /Stf^y FIHDOBJ and jf 
CON-\TRT are used in graisaar-produced 'questions, while Jser-initiated ' * " 
J questions nay use FINDbRAW and .IWFO. S. • ' V . 

QASK is activated by the LA Controller after tk parser returns with 
^ • an indication that', it parsed a question form, ^s fil^tion must" determiiie 
how to answer i^user's question based on register settings created dufing. 
th§ parse. The cpllection of regist.ers is passed as the function's' one 
argument. If the user 's .answer has not already been determined djjring 

Parse^th^ an appropriate message is sent to retrieve additional . ~\ 
» information from QA. •'••<;. | 

J- The function FINDOBJ is used in d^ermining the' reference of^-name" ^ 
mentioned in the input string. Periphrases like '"the point P" it answers " ] 
. the questions "Is the oUje^n^^d P a poiiJt?:' and "What is the node I 
y idehtif iet in 4=emory for the^nt nased P?" The ,parser calls on this J j 

function whenever i'nahe is mentioned.- This is espteially useful in * '' ' 

ERjC • .20 ' . ^ \ .v.] 
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det^ing hdw a word is being used when no lexical' entr^. can 1,e found. 

Conversions to system uaics-'are accomplished through the CONVERT 
feature pf QA. The only .units 'reco^ized i^ MG are' sJre'en units fof « 
lengths and^.degrees for angles. Any other scale suit be concerted to * 
these and the gramar contains appropriate calls to QA to accoinplish thlb 

^I^^^KAW is used- to determine if a, particular e-^tity is drawable^ 

or erasable, and to retrieve the list of objects.drawable by the system. 

Its use fs linited to answering specific questions lii^ "Can you draw 
or "What can you draw?" ' . \ 

" . Thie iKFO feature of QA encompasses ^11 .other types of quastions.-- ' 
A^prototype structure, similar to that built as a final f orm o'f ' 
representation for completed sentences, is partially filled in with the ' 
information known at the time%f the call and passed as -a para^ter. la 
•addition, a list of the c.ses that need to be filled INFO -is also^ 
passed. rne^resp<fnse arrives in the form of "a prototype with the requested 
cafees being filled\f possible, if n.re than one item can fill a casc' 
slot.^..en all of these are returned. For example, in answering a question 
IlKe What did^ou draw here <T>r' the question is put to INFO in the form 

Clve^me the %OBJECT and %KAKE of the items drawn at <T>.". . response " 
is lat.r passed along to GQJ so that the event<ial 'answer comes out as 



for example; 



"a eiSCLE named BALL" 
"a POINT named FRED" 



3.6 Problems LA ; • ' ^' 

>A number of minor problems* continue to be a pol^t of concern in the' 
language ^alyzer. Of course.^ it« is difficult to assesl the ulti^te limita 
tlpns of the approach taken here, but hopefully no major obstacles will 
be encountered in the inraediate futur, to impede ^the «ttendability of the 
system. A few problems are discussed below. . / 

The adi^^ives "small" and ''large;' should be handled heuristicaily/ . 



thi^, the object must be considered and appropriate size cal- 



In ^oing 

cula^iond mast be made relative to its shape. .The' current 'app^I^^^'^s 
adequate guess for average, s'li^e circles on the screen J- but' a "^one 




:er" small circle and a "four iqch dlaineter" large cir;ie 'may 
not ^ways be appropriate. This ^robl4m is not .unique to these tw^ words. 
Consider, for ex^.«ple, how to bandit "near" or other locatives such as'' 
•above", 'tbeW,- "behind", or "ne«: to" • 

. ^ ■ .21 . • • ^ J 
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^ Prepositional phrases containing "to" and "from" are handled. 

:identically. They each result in creating. an ENDPOIKT tag**tr the 2P0SIT 
■ case of the prototype. Thus,x;the direoxionality of the .statement is 
lost. Such a' distinction was deliberately not made since a problem 
can arise ip the graphics routines iya line is drawn in one direction 
and an erasure is attempted in theV^ther. Some simple solutions to the 
"endpoint" problem exist and should be forthcoming. ' 

In parsing, the graiaaar should perform more consistency^ tests 
» Hith objects and their parameters. Currently, the graiaaar 'allows input 
sentences to specify circles drawn at ahgles, lines drawn with circum- 
ferences, and points to .be drawn, "from here tm there". 

The- prototype itself may not be eaisffly, extended t>o handle, for 
exanple, a "move" operation. One manner of specif icatlon in a move 
could indicate an origin and a destination position. This might require 
i|troducing subleases in|o tli^ %POSIT case. Other new operations my 
require similar extensions! 

The verb largely detenaines what structure gets built during parsing, 
^ There- ma/ be no easy way out of this, but to extend th^ systeja to handle 
somar feimple Elliptical forms like ''A circle here <T> please" one oay want 
a isore flexible scheme for determining the prototype. 

^- THE LANGUAGE GEKERAIIOK MODULE ' ' ■ 
' ■ -J ' J*-"^ Function and Overall Design _ ^ 

^The generation of language responses ,to the user in the system is 
^ ■ performed by the Language Generation module (GEN) . A smaH 

collection«of j;emplates serves as the basis for generation (see Appendix 
\ ^ L) . A module can request that a tespouse be generated by specifying 

a template number (TO, Tl, T2, ...) and an or'dered list of substitutions' to 
• . be madfe for the asterisks which ^pear in the template. 

j ' Besides this straightforward method, there is another way. that responses 
can get generated In the system. Instpad of Indicating thetemplate 
number and substitution lisr as the two ai;gument^ In messages sent to GEH, 
S^essage may contain the indicator GENANSl pr GERANS followed bv a 

' / ■ r. . • 

prototype. This Indicates that either one' or i3any_answers are required i 
t • from the prototype given. A generative m grammar processes these 
requests by parsing. the prototype and producing a list of cemplate 
numbers and substitution lists, one f<Jt each' response . The graiaaar ' ' 
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i 

at- this stage of developoent is rather simple '(se^Appendlx M) but ^ows 
.promise for future work. ' . ■ ' 

The 'techniques^ used in the GEN module work adequately for RLG as it 
now exists. • is hoped that as improvements evolvi Sn other modules of 
the system, GEN will proved be expandable as well. 



r 
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5- THE KNOWLEDGE BASE MOPm.R 

•5.1 Function and Overall Pesigfe * « , 

J XXU »dul. ..cepcs a .easag. Passed to It ft„„ ' 

pa t t tT - = ^° 

part th, arsu».„t slot of the ...ajge. Control is passed to the 

ctloo (a ^ ^^^^^ ^ 

is 71C words. The nodule has the following tun.tloos: 

1. to aosver questions about the s'tored- tnowjedge posed by 
other.oodules In the systeW?-, " 
II. to b-ulld, erase, and change.set^ntlc Lory structures. 
111. to produce a -essage to be sen-^ to the Graphic, «,d„le 
. »hlch will co„rrol the drawing of an object on the scr;en. 

5.2 The Semantric Merory / * ' ' 

5.2.1 St^ructure 

« 

dlscu T"' "-'"^ ^-"-"d by those 

cussed m Brach^n 11976,. - They are manipulated and built u.lng a 
lightly «dlfled version of HEKTAL (Shapiro 19U,. The netvor. Is used ' 
-encode both th, system.'s prior ^ovledge about objects and data if 
^ jects introduced during the discourse. Attac.i, to the descript ons ■ 
Of Objects m the netvor. are actions (ACXs, which are encoded In US 
used for checking structures, producing p.rt »f the output oessage 

r gi:eTa"':sVTT"°° " °^ — 

,» gives a list of the actions and their function., the Table below 
g^ves a list of the labels used on the Se^ntlc Hetwor. links. whUe 
*pendl,Vdescrlbes the Initial. state Of the network.- 
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LABEL 
J- SUBSETOF 
ELMNTOF * . 

TYPE 

'J ♦ 

ELMNTDESCR - 
DATTR 

ROLE 
RESTR 

ACTION 

ATTR ' 

IHSTltCOF 
VAL 

NUMBER 

STRUCTURAL 

INSTANTIATES 

OBJECT \ 
ERASED 



TABLE OF LABELS FOR SEMAJTTIC NETWORK 

\ 

STANDS -FOR ^ . 
subset of 
Element of 
Has name 

Has ^pe 

- Element description 

Descriptiop of attribute 

Plays role 

Is restricted to 

to action 

Has attribute 

Is an instance of 
flas value 



Number of occurrences 



to a structural check 



Instantiates . 



to .erased vobject 



LINKS.^ 

^ , COMMENT 

connects two sets ' 

connects^ it,em to a set* 

for item, iSf: set,, or 
coacept " «» 

1 - .^u- : 

for* every node (SET, 
CONCEPT,' ITEM, DESCRIP- 
HON, ASSERTION, SYSTEM) 

connects set to descrip- 
tion of a typical element 

from concept node to 
description 

from description node 

connects description 
'to set 

^ frotn description node 
to action name 

connects item to 
assertion 

connects item to coh/:ept 

connects assertion to 
item^or value ^ 

connects description 
to a number 

connects concept to 
an action 

connects assertion 
to descript^n 

in simple screen model 



lie ^i|ckbone of the netJork is a hierarchy of sets, including straight 
lines^poifiis,.angl|^, distances, defined points (i.e.. locations bn the 
screen; nof necessar^jr illuminated)', and circles (Figure kB-2). 

Consider the portion of the network which stores^rior information 
about circles (Figure KB-3) . *— - 

Kode 1 represents the set of all Circles. . Node 2 represent! the% ^ 
description of a typical element of that^^et. Node 3 represents one fif the 
parts of the description. The^art shown hei-e is that of ' "Locat^ 



Vhich is associated with the %POSIT case in* tfie input prototype, and " 
rfstrifcts things which play the rale of Ipcator to oenibers of, the set pf 
defined points (Node A).' The ACTa wilfbe^jJliified later.' ' 

Figure KB,4 ghow?, as a result of a .request to draw on the ^creen-^ - - 
how information about an object Is^ at^re^ in the network attached t6 the 
description of that type of object. -Subparts of the'Sbjedf ;re dlso 
connected in this fashion/ Notice ^he "assertions" in the network.' which ' 
include -the asg^t ibn"(node 2) that the locator of- the iteiS (node 1) is 
a^partlcular- defined point (node -3)". 

5.2.2 Actions ' ' . ' fT 

. ^^''^ ^""^ '^^'"^ types of actions attached to the network: ♦the 

^ /> Structural action, the Default action, and tbe Todraw action. . 

The^ 8tl:ur^ural act is nised to check pieces of network after they have ' ■ 
been built to the spekficat^on given in th^ input pro^otv'pe.- For example 
. the net built for the" new circle in figure KB-A ls*kissing si?e infor- 
mation (i.e.. Diameter, Radius, or Circumference). ACT-19 (see Figure ^ 
KB-3) will detect this omission and will report th^'t'the role of Radius 
^has not been instantiated! In general the structura^ act 'knows* what 
^ . . • the minimum requirements are for drawing^ the A^bject , ihd will ^turk' a " 
list of those missing. In the case of a circle, the requirements are 
Locator and Radius. ' ' 

The Default acts are activated if a non-null list is returned by a 
Structural act.. In this case the act used is the one associated with 
. . the node which has a Role link j:o RadiJs (i.e., ACT-22; see Figure KB-3), ' 
The Default act, will .look to see whether there is anything in the network 
which will allow the missing value to be calculated (e.g.. 'r^Ius calculated 
from Diameter), -and if not, will provide.a reasonable default ^&lue . .When 
'it has a value, it builds an additional piece qf network' and 'coiiq,letes 
the requirements of the Structural ^ct. A Default action is run for eV^ry 
••• omission. the impl'emented system only circles have default actios . ' 
The Locator defaults 'to a touch point slightly tq the right of the center " " 
. of the screen, and the Radius defaults to 100 scto^^n 'unite (approximately l.S " 
~ - * ^ inches) . - • . • , • 



t '• '■ 1 * 

liyt^.^^^^^'^l manylmoortant links^are not shoya in Figure KB-4.- ' Noie 
that t^is is the^form of the network prior to defaults^ . 
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Dnce the item Is^ fully specified the Todraw action cau be used. 
Ic takes the^ piece of- nefcwor}t ^representing the item-to be drawn and 
f>roducel from it a program fprm wKich will do the job. For examole, the 
circle, with ,lfes center at the point (100, 200), and its defaulted" 
radius of 100 uni^.^-will produce (F4CRCL 100 200=100). ^or a straight' 
line thp Todraw act is'a- J.ittlfe more complicated as it has; a phoice • 
which depends on which comb.ittatiory,of endpoint^ midpoint angle, and length 
has been specified.^ '(See' Appendix P) . ^ 

5.2^3 Naming and Erasing . . - * 

Tfie userjhas the ability to ask for an object to be drawn, and name 
it, feoth in the same sentence (e.g., draw a circle ca^ed Ball at (100,200)) 

results in a ffam| link from^the item as in" Figure KB-4. If a name 
As not specified thpn the system provides one by taking the digits fi^m 
the MENTAL node identifier (e.g., N2345) and appending them to the first-.two 
letters of the conoep,t name (e.g., CI23A5) . ^ery node generated by 
MENTAL Is automatically given a unique identifier. * 

When the user, re^quests that an object be eoased f/^6m'the screen the 
syst^ leaves? it in the 'network, but marks it as erased. This acts as a 
simi^le screen- toodei, and prevents the system 'from attempting to erase an 
object that is^ already erased. ""(See Figure KB-5). 

5*3 Question Answering - ' 

5.3.1 Ititroduction - 

This section of the Knowledge Base is Activated when another oodiae 
needs some infonaation. The syntax of the* various QA calls is given in 
Appendix Q. There are fo^ufv available functions: 

^ i. IN^ - A genetal network searching function,^ and the 
;* n»st po^/erful function'of the four, ||^ 

f 11^ FINDOBJ Os.ed for disco^rering whether a word whichlls 

suspecCed of being the naine of an object acturily ^ 
n a me s some object cm the screen. 
. iii. CON^RT r Tbis 'is a function for converting lenj^ths \o ' 

screen ubits or anj^es Into degrees, 
iv. .PINDDRAW ^ - ^ Used Tor find2ng ou^ whether an object can be^' 
\ drawp; 

5, 3, 2 The INFO Function ^ ^ 

*. A prototj^pe. form and, a list of ke^ords are passed as the two / 



parameters of INFO. The procotype Jorm is used to search for items 
'in the network. A list oT items hatching \he description (i.e., the 
prototype is. viewed as a description) is obtained and those items are 
further searched for Vao. information requested by the list of keywords, 
For example:' 4^ 

, ■ (INFO (%bBJECT WAME . NODE ZPOSIT) ' ' '■ 

^ ( (%POSIT .( (LOCATOR Tlt)0. 200.) ) •). ) 

))'.;.. ' • " . 

could give : . ' \ 



( (%OBJECT . (CIRCLE POINT) ) " \ 

(%NAME- ..• (BALL FRED) ) ' i. / 

, • • (NCtoE . fN2345' N3442) ) . '\ 

(ZPOSIT .-4 (100.200) ) ■) 

i.e., a list of dotted pairs of the form (keyword . list). The answer 
to the question "Which^bjects are at (100, 200)" therefore , is a circle 
^^^oed Ball and a point~2alled FRED. " 

The INFO function is capable of handling positions specified by 
. LOCATOR, ENDPOINT, and TOUCH. . The latter will successfully match both ' 
LOCATOR and ENDPOINT'during the search, but preference will be given to 
LOCATOR. Corifesquently, if the new inputs "Wha't^did you draw here <touch>?", 
and the touch point corresponds to the endpoint of a line and the center 
of a circle then the response will name just the circle. However, if <he 
touch corresponds to two LOCATORS (or two EIJDPOINTS) then both will be 
returned. 

The function uses the'^TEJiFIND procedure which searches' for all > , 
objects which- match a gli?en prototype. This -is done by considering each 
of the cases in the prototype separately (e.g., ZKAME, %SIZE), obtaining 
ysts of matched items, an\i Chen forming the intersection of those lists. 
ihi resulting liat gives the ITEM nodes of all objects which possess all 
of the required characteristics. 

Tl^ere are two error cjsnditlods for thfe INFO functions. If there ^is ' 
no ofe/ect that corresponds to- the description .then the message "object 
as specified cannot be fouad" is dutput to the user. In tliis case, and 
in the case where the object is, founiJ but there is no information corres- 
ponding to- the given-«keyw6rd, the "liflt" returned with the Keyword is 
NIL, e.g., ((%POSIT% NIL)). " ' - ■ • 
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'5.3.3 The FIN^BJ Function 

Thete are two ways to use this funct*f»n. The first is to. specify 
both a suspected naae^ana an ob^'ect type in^the call, e.g,» (FINDOBJ BALL CJECLE) 
This tests the netvork to see if theri^ is an object of the spec d^f^ed 'type 
with-^e 'specified tiaoe. If tljete is, then the node identifier of that ' 
. item^ retumea", e.g., ((KODE-, N234^ CIRCLE ) . If there Is no 

op^t of that type with that liairs, theS KIL is returned, e.g., (NIL CIRCLE). • 
If, by accident,. an invalid obje?^ naoe h?s beeii given in the call, then . 

is returned instead, e.g., (h^ NILX The isecond^type of call allovs 
the object type not to be specified in the calj^ (i.e.T KIL is_uSed>. Tnis 
works £Vi the sace way as described abqye, gzcept that 6n return it 
inserts the Vabject type of the itec with the n^-e given in the call, 
e.g., (fLHDOBJ BALL NIL) gives (( (HODE . H23A5) CIRCLE )). 

5.3,4 Tne COI^RT Function 

rne response froa this function is the valjje obtained after" converting 
the given nunber into screeti units, or degrees, depending on the unit • 
specified. e.g.,f(COKVERt 3 INCH) gives" (192). Ik>te that the screen is 
8 inches square, add 512 by '512 screen units. 

' '5.3.5 The FLHDDRAW Funcelon 

• « ^ * 

• ' There are two use| for tbis^function.' Tne first al0?vs the question 
"can you^aw an X?" to be as&ed by using zhe call (FIHDDP>AW X) . The ) 
response wiU be either T or KIL, for yes "or no, re^ectively. Tne second 

^ use is to aijswer the question "what can you draw?", e.g., (FIEDDEAS KIL) ' 
X gives (SI-LINES POINTS CIRCLES). This function operated, by arching the 

nywork for'^^cepts with a TODP^ role. It -is assi^ed that as Vhey can ' 

be drawn, then they can. also bp nased or erased. / 

5.4 Kemry and Graphics • .. _ 

5.4.1 Introduction ^ , ' ' , . - 

* On . receipt ofvthe oessage, the p'iK>dule ^s^srsines"^^ first itc? 
of the arguoent list. If it is 'iS;" 'the Kes»ry and Graphics section 
takes- control. Ihe mrPE value is esaained next and control pass^ to, the 
appropriate section depending on its value. Bote that only ii^faCives 
<LMP) are iaplecented .in' m. Next the XAciim value Is used to switch 

^ .-to the NAME, gEASE, or DRAl?'"secti^ of KB. 
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5.4.2 Drawing . . . ' ^ ' 

* After 4iptfliJ^ the set and ?on<5p^^s for t^e object t<i.be ' 
drawn, an. ttea node is Inserted into the network fo^is new object, ' 
and links are fo;rDed 'between it and'the^set aj^^i ' c^icTpt nodes. If a 
nace is given^ (e.g. . »i>rawa circle called Ball") this is added to the itea 
node, othervise a aaofe is generated by the ^ystea as previously described. 
SThe orientation, position, SBd size information is exacted in ttim V 
/ro= the prototype, and appropriate AT^m and n^TAKTIATES links are 
set up. Kote that the positional inforcation nay ineinde a 
{!WDE . <aode identifier^) fora.'due to sentences guch as "connect P 
with VlOO, 200) vhere^P is the "naxr« of a point, and the node. identifier is 
that of the ite= node for the point.- 

"ne Structural act is used. to check. the structure, as previously 
described, and if necessary, default values are inserted in 'the network. 
-Once the net for the ite= is properly specified, the fODEAW action .is used 
to produce a progry foni for inclusion in 1 lues'sage which is then sent 
to the .GR Ev>dul«.-4his message contal^is a a>dif^ed fgrn of eije prototype* 
chat was sent to the K3 KKiule (Append^ ?)'. After the dravi&g is 
corpleted, the screen -adei is adjusted, aad a oes^ge is sent to .the 
IK!T ^ule to fadicate successful cocpletfon. c ♦ - - 7 

■ I ^ - - - 

3.4.3 Rasing ... 

If the s'creea is to, be eras.a*r-tben tne screen o>del is^tered to 
cark all iteiis as erased, and a message is sent, to the GR i»dule to erase/ 
-the screen. On return froc GE, tl^ m rsKTule' itself sends a success^ 
conjfcetion message' If an object is io be erased, and if that object has' 
not bee^ directly specified, theajlj^^twork is searched for aU ite^ 
that eaten the prototype'. If i^ than oxy/is ftimd the itecs are checked 
and -all those which have already been erased by the user are deleted from 
the list, i'roa the itesss reaaini^g, one is chosen to be erased. If the 
object has been specified then the fora (BODE . <'m>de identif^>) wUr 
be use4. Ibis reaoves the necessity for any further. Searching, ilM 
object can be erased directly. The TODRAH act is used to produce a pro^aa - 



•^te that the positional in'forsation say include TOCCH fonss. ^liese ^ 
wlU catch both EHDPOirr shd LOCATOR during the search. '^'^ ' ' 
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forp which describes the erasure. This is included, in a message 
which is. sent Co GR^ The only difference between ^raw and erase 
message for the sane object i^ that *the 2ACTI0N is narked as DRAW or 
^"E, respectively. Consequently the prograa fona can be produced 
W exactly the sane Todraw action for both operations. The GR module 
^- de\ects- the difference in the |?rototype and acts accordingly. After the 

erasing ii conpleted. the screen «>del is adjusted to aarrthat item as 
y erased, and. a message is sent to the IKIT Dodule tq indicate successful 
camlet ion. * 

5.4.4 Naning 

% 

Note that -Wng- in this systen' refers to the naning of objects 
^ravn on the screen, and does not nean the naning of groups of Objects 
(e.g. ..four straight lines being calLtd a square). T^e fir^t action is to 
reoove the (%NAME . ■<na:^>) pair, fron the orototvoe. aid then use the 
new prototype to search for all itens that =atch that description. 
Here, as before, the positional inforastion oay Include TOUCH fonas. 
rnose that already have user given naces are discarded, and one iteo'is 
picked froD the resulting list. That Iten node is deleted fron the net " 
and a new node is inserted having the same links, except for the na6e 
: link which points to the new naee! , Fiaaily a ^ssage is/senb to the 
INIT aodule to indicate successful conpletion. 

.:>-:> Protlens and Extensions- 

5.5.1 Moving " , , ' ' 

rnere are three types of operations of this type that we wuld ' 
consider adding to the systen*^ ' • ' , ' 

1. Draw ifa the sajse place with no change after a prior eraiure, 
e.g.. "Draw U." (a null sove) 

_ il. Draw tb a different place with no erasure of old object. 

e.g.;, "Copy the circle here <touch>." ^ ' , 

, ' iii. Draw in a different place with erasure of ti^e old object. 

e.g., "Move LI to "here <td6cb>." • ' 

It would be Slice if Ihe aechanisn which.handles these could als'^ provide 
changes -in the values 5f other cases on request (e.g.. "ihrawU here but ' 
at 45 degrees and 'an inch saallei^'). 'hany of theJe details have ^,een. 
worked out, but have not been ii^^leoented. One reasOT for not lapleaenting 
thitf w^our suspicion that' In the' vorsjt cases tte prototype might need ' 
'slight airisions. -To do the sbove operations at all requires special ' ' " 



29 



interpre'tation of the, prototype in orde?~rb be able to specify the existing 
objept being referred to, as well the new reqwirenents. 

5.5.2 Remembering 

4 * 

^ We considered implementing an action that would ailow the user to 
•rteeober' the obj4ct8 that he had drawn by having all or part of the 
s.eoantic net dumped to a file. A 'retail' function could be'used to 
read back information fron specified files" (i.e.. those created b^^he 
remember function) -and re-establish it as the semantic net. This- pair 
of actions could be use^ j:o maintain, a' record of the state of *an interaction, 
and to save complete or partially completed drawings -between sessions. ^ 
jlemembered pictures could also be displayed, after appropriate preproceCing,^ 
on other devices such as a plotter or^a CRT. In addition, it may be 
useful'-to have a 'forget' function which could be used to permanently 
erase pet structures representing all, or part, o^ some .item or items. 
This could be us^'if a mistake had been made, or if ^ome object wis 
no longer required. It could be argued that the erase action should 
remove the. item from' the network. However, 'this, would remo've the possibility 
of redaawing a named object after it had been erased. 

5.5.3 Defaults , * 

A simple extension of the default mechanism'would be to mark all ' ' 
pieces of the net which were built a^s a- result of default acts. This 
would allow Che system to recognize the difference, and enable the system v 
to remind the-»user. of exactly what he ^ad specified for an object. In ' 
^addition, defaults could be provided for ail of the primitives, whereas- ' 
at present, only circles have default actiofis. The only additions 
required would bq to slot^the default and structural acts into the code for 
K^.and create^ links from tJfe appropriate nSd^ of the concepts to -the 
act names. The activation mechanism will Vork for all def^ul.ts in the saoe 
way. ' < 

-Structural acts return a list of those roles which need'* to be iii^tan- 
tlated before, the Todfaw act caiT function correctly. It may T)e case 
that the Default actions should be executed in a preferred order so as to 
make maximum use of the infon^tion^giv^n and to minimize the V«aber"o£ 
defaults. ' . , . 

For a better treatment of Jefaults, two extra tfectoiques would- have 
to be Bsed. The present, iaplei'entatlon of default's is esgentially context- 
free, and merely u^es built-in values. -There will be situations %men this 
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is aoc sufficient, and where a heuristi^^default oechanisin woulxl be - . : 
appropriate. For example^ a default action: could execute a space-finding 
routine to find a reasonable location for the object on the screen. In ' ' ' 
other situations it would be sensible to ask the user for the extra 
^ -infonnation. if,. for example, the heuristic. fail^ to.-find *a suitable vaL. 

5.5.4 Structural Information - !- , 

There are strong grounds fort arguing rhat the Structural act 
contains 4:oo .much infonnation, even for such sii^ple it"eas as point,' line, 
and circle. This information, for example what ful^ly defines a straight ' 
line, is -not accessible as it/is encoded in the LISP code. For even only ' 
slightly mor^ complex objects, such as square, it appears that this infor- 
mation should be encoded in a network form. This would allow a new 
of'QA by LA. During parsing, to aid izi establishing the role of prepositions 
it wetild be useful to have-LA aski QA what other cases to .input, given the 
current input. ^For example, having found an angle and endpoint. and son»e- ' 
^ thing else wkch is not easily pa^. QA would be able to suggest '. 

trying to look for a.length descrip>^,il^tructural. network would pl^ * 
the important role o£ specifying t{,e relationships between the various ' 
attribute descriptions. " • ' 

5.5.5 Defined Points ' • ^ . 

^Ac prese«t, whenever a defied point 'is specified, no attfempt 

is made t^detemine whether there is already a 'structure, for that 
. point. A simplOitension of KB would be ^otnclude a routine to check ' 
, for existing structures'. TMs would mean that lines' whic^ shared 'the 
5r89'?e end-point would share the same Defpoint structure. 

5r:^6 Point, Notation *^ - ^ ^ . 

,- It -might be possible to use a unified notation for points, lines, 

and areas. For example: * ' ■' • 

a- particular point . <100, 200> 

the line y - 200 - <vx, 26o> 

any point, with y - 2(* <? , 200>- ' 
^ " any point " <? , ?> 

the screen * <yx* 4y> ' 

a hprizontal line segment <50 - 70, 200> 

a rectangle. , <5o _ 70, 100 - 1S0> , » 

any rectangle <xl - jeZ.-Vi ->2> — ' 

In a more sophisticated system it aight be useSful to have volumes as 

, B^rlmitives and other basic objects* as degraded voltes. 
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5.5.7 Screen Model and Action Sfetory 

In order to handle words such as inside^ and outside (e.g., 
"please drawa circle inside the triangle") ^if is necessary for 'KB to have 
. good knowledge of what is on the screen. For 'outside ',. for example, it 
isjiecpssary to find space for" the object to be drawfa. The stored 
knowledge about items in the -semantic net in some way provides tTiis 
information, but space-finding could involve, inspecting every item, and 
. much calculation. Consequently we would prefer to have either special 
hardware to tell KB which lines or points are illuminated, or soke sort, 
of comprehensive model of the screen, possibly in a bit-map form. Such 
a model could be used for space-finding, and discovering spatial relation- 
shipV and topological properties. . 

To xieai with questions-^ Lhe^JTind . [^'What did you do?" and 'Vhy did 
you do?'% an action Mstory s^jould be included in the system,^ logically 
in the KB module. This would consist of a time opdered set ©Tactions ' 
with the^f'^ociated details. Use could be made of the simple indexing 
meohanism. available through set^membership links. These'would point to the 
set of all objects'of the same type, and ^hey in turn would be linked via 
a "time line". This sort of mechanism could also be used to help with 
anaphoric reference. 

5.5,8 Functions 

For positional indications such as "near", "at'U or "center" wel' 
will probably need to have words that trigger functions in the KB.-^i^ha. 
This has not been carefully worked out yet, but we feel that forms sucit 
as (%POSIT . (NEAR (TOP #SOREEN)) ) or (LEFT (UPPER #SCREEN)) could be 
used, in line with our general principal that as much information as 
possible should be extracted from a sentence before KB gets to act on 
it, we feel that QA .would probably b^ the correct place to implement 
such function handling^ This Uould"^|p^|^ LA to pass areas or coordinates ^ 
^ to the KG section* . _^ - . * 

6. raE_ GRAPHICS MOD.ULE _ 

6.1 Introductftn , . ' - 

_ This module accepts, messages containing screen alteration comands 
and executes them. FJ*lt th'^^essaga. is inspected to see what the Action 
is, and a marker is set depending pn whether it is draw or erase. Then the 
argument part o^ the message is executed one step at a time. In the 
cutrent system o'nly one step is present in each message (see Appendix P ' 
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for GR message foribat). The market se'ts the underlying graphics taachine 
code routines into either draw or erase mode.- An object is erased by 
drawing over it 'in erase mode. 

The development of a LISP/Blasma. panel .package (PIASUB) interface 
• enabled us to program the module entirely in LISP. -The interface also " 
allowed the use of trigonometric routines" from the FORTRAN Library, and 
consequently we were able to do the angular . calculations with ^se. The 
- reason £or the F4 prefix on the drawing function- names is that at one time 
these were to be written in FORTRAN. 

6.2 The Screen Functions ^ 

The draw/erase functions consist of -point drawing, circle drawing, 
screen.erase. and line drawing routines. There are four line drawing 
routines, one for each sensible combination of the two endpoints. the 
midpoint, an angle% or a i^h. The point drawing routine uses the 
increment mode of drawing to draw a point consisting of ^our dots. The 
line routine uses.a basic vector' operation. Tne circle routine uses short 
chord vectors to produce the circle, with the angle ste^ size that defines 
the chord varying depending on the size of the circle required. 

The PIASUB package routines us^ in GR.are: 

ERASE - screen erase 

SETMOD - set mode of panel 

INCRE - incremental draw/erase 

VECTOR - linear draw/erase/move 

Working with the plasma panel involved us vit/ three different- . 
coordinate systeias. 
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6.3 Problems "and Extensions 

Thi circle routine is less thin perfect at the ooment, but the 
authors are aware, that. there are better circle drawing techniques that 
could be used if necassary. Because of the large number of vectors 
used in drawing a circle, a^d because o.f their frequency, the circle 
drawing routine suffers occasio;tally=»f rJ>m transaission errors, as the 
panel is connected remotely to the DECSy^tem 10. It may be that using 
increment n»de for circles would improve* this . 
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7. CONCLUSION , ^ ' . * 

7,.l System Performance 

7.1.1 General ^ 

<> 

The system responds fairly quickly to user input, with the onl^delay 
being due to the fact that the terminal -1^ remote from the computer and not 
on the fastest possible line. In consequence there is a substantial wait while - 
circles arc being drawn, aggrevated by a slightly inefficient-circle drawing ' 
routine and by being swapped-out occasionally. The sentence. processing how- 
ever, runs in approximately real time, with an average processing tipe of| well . 
under a second from input^to prototype formation. A slight speed-up of the ^ 
system could be obtained by compiling thje KB module-of the system. This yas 
not compiled due to the fact that a large number of structural changes would • 
have had to be made before compilation cOuld take place. The average pro- 
cessing ti^e for each sentence from input to completion is about 2 s'econds, and 
depends on sentence type, sentence length, and screen operation. The number 
of sentences that can be put on the screen is about fifteen, and the nuaber 
oF KB-structure-building sentences Cdrawing coimnands) that can be processed * 
without running out of 3pace is about. ten. It should b6/noted that as a 
spinoff from this project we now have a LISP-FORTRAN interface which may be ' 
used for other LISP projects involving ' graphics. , A version of LOGO h^s al- 
ready been programmed using this interface. 

7.1.2 Good Things 

The technique of modular construction using .message passing and a ' 
control executive proved to be very successful ^^_We were able to dWelop weU 
defined interfaces, and pursue development of Mkes individually using 
module test;ing' routines. Modules were initially as "dummies" .and 

;graduklly built up to work 'correctly. We were able '§ use dummy modules to ' 
build a complete system inciting, but isolating- a nS»ly {fhanged module. 
The ability to monitor the messages in the executive allowed us to "diagnose . 
interfacing problems as well as errors wlthtn modules. 

We were pleased with the relative ease with whicK^it was possiblfe to con^t^ 
the system, given the use of the ATN and-Ji^AL packages, and with the Wledge 
of a "handful- of AI techniques. We take this to be an -ifidication of the ...^ 
progress of AI technology. - * 
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. . ^ Despite the limited grammar, and the relatively simple defaults and ^ 
^^ystem-heuristfcs. 'inte^^acting with the syst'em was pleasant, and its "in- ' 
. telligence" was very' convincing . Several people who ^d had some^Computer 
. Graphics experiende^ reacted favorably to»dem^6±rationS of the system, and " 
were incline3 Us everu*^ were sometimes) .to ^ibe the system, wit^ greater 
^ intelligent than it was. capable of displaying*. , ' «' ' 

^ ml ability to touc-h the screen with one's finger proved to be a use- 

^ful addition to the stilted fofms of lan'gua^e- normally investigated in . 
current research on Natural Language Understanding, Systems. The processing 
of deictic e^tfpres^ions. including^ce adverbs and demonstratives, followed • 
difectly once touches were^ncluded. . dl 

Tha.^us^of the INFa function in the QA sectioa of KB allowed the .user to 
specify objects by using" a partial description. We feel that this is a power- 
ful technique, and contributes a great deal to'' the ease. of interaction with 
. the system, » 

7.1.3 Bad Things , ' • ^ 

■ - 

., Ap overridin^roblem with the system was. that, "despite our beft'-ef forts 
it tended [to gteow on its own: Some unplannedlfor problems ineviVbly^ccurred, 
and .were solved,. but due' to constraints on tide and effort were not necessarily 
solved in the best way. As usual, ther^^.are bijLS we would like to recite.* * 

It is not clear whether thrslmplistic wcrld of l-ines, points,, and circles 
truly provides a basis fer asser tions_^about more complicated worldsv In ■ " , 
Addition, we^were a little supri«j^fiat even with just line's, points, and ^ 
circles, and drawing and «;as^i/g, the system had to deal with ^±te com^pli^d 
^ situations.. This may be partly a ref I'ection^f W own'naivetl, and part Jj. a 
. strong indication thaf^en the simplest of AI systems should not be under-'" 
estimated. # / . -C 

It'-wDuld'^e difficult for us to claim tl^ our work .resu^t^d in a 
practical system. From a hardware, point ofii^'. a major weakness is that 'v 
■ j:he touc^ panel does .not allow very^fine resolution, and consequently, precise 
iplacemenL of objects must. be done using coordinates. 'The system also lacks 
many of ^he things that one would expect in. a Computer Graphics system', how- . 
' ever, this is to be expected as it was never our' intention to Implement all 
of tl^e facilities in the pilot^jiroject . ^ • • , 
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More extensive modes of graphical input would certainly enhance 'the 
communication in a graphics enviror^nent. Specifically, one oft.en would 'like 
to simply sketch out an/object rather than verbally "describe it. Develop- 
ment of a representation which would model the esgential ^racteristics of " 
sketches would greatly aid such an effort. ' . * ' 

The physical management of the -screen needs some Improvement. Since the 
user is permitted to draw anywhere on the ^creen, text and pictures often over- 
lap. Providing a division in the'screen between the^ drawing area and the ' 
area for text seems to be the. best solution. The text area could be at^ the 
bottom of the screen showing only the last input and response. ' ' ■ 

■The limited syntax of the pilot system , is too 'prohibitive . A response 
of "please rephrase" doesn't provide the user with many clues to 'what was 
wrong with his input [Weischedel 1977]. Certainly, a'd 'intelligent system ' 
would at least be able to say that-it had ^o knowledge of squares if aske^ to 
draw one. Likewise, if the input contained all o| the essential parts tb 
specif y 'drawing^ an object, the' grammar didn't p^t the form, the system' 
^^uld override the grammar. lK ^' 

We felt that as a tool for matiipulating knowleaiPMENTAL was "^at too low 
a level. In addition to this, the Brachman-like formalism adopted for KB 
uses falrfy complex structured. Thi's-meant thaf a sl^nj^e insert or change 
involve2y several operations. If we, could start a^ain. we would build >a higher 
lev^l of language over.MENTAL and ase that for net manipulation [Bobrow 1976] . 
We are aware that MENTAL has evolved, and that newer versions may be significantly 
easier to work with [Shapiro';1976] / - * 

^2 Outstanding PrablCT^- ^' ' - , 

. Unguage generation, as it exists in thfe pilot NLG system, is quite 
weak. AlthoQgh the foundation has been laid Jor developing', this module, little 
effort has been given to this. Nevertheless, some very simple mechanisms hav€ 
.proved .quite ef f ective-'in allowing an adequate , level of co™u;ication. Lan- 
guage is generated u^r the gufdance.of an ATN generative network whi<:h ^ 
8uggests^.a template and a list of substitutions. Thus, output is limited in 
form to j^restored templates. This requires that all system responses be antici-' 
pated to some extent when the templates -ar«, designed. Aiso." the knowledge base 
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reolains inactive during generation, unlike in language analysis', should 
be ai>le to use the QAJnterf^ce to' obtain additional infonnktion to use during 
generation, and it should also have access to the lexicon used -by U. ^ 
• , There are a number of interesting but outstanding problems connected with 
. the KB module. The first has already been alluded to slight]^: that of \ 
developing a full'action history aod model-of the screen. The ability to refer' 
to past actions and their ordering in tiae would ali^ a^whole new'range of - 
. questions to be asked. It would also^elp with reference problems, as in. - ) 
for example. "Now connect the other ends", or "Draw another". The screen 
model, if it can in fact be ^llt. would allow heuristic , placing "of objects" 
and use of heuristics such as those for "near", or "beside" without undue ^' 
caleulaticm* . . . " 

we believe that if knowledge is to be- collected together ^t all in ^ 
systa.^ then as muct, knowledge as p1,ssible :should be included so that rich Inter- 
connections may be established. It is Reasonable therefore, to include lexical 
information in KB along with descriptions. For example, the word "CIRCLE" 
and its associated markers could b. stored with the ^oncept of circle in some 
way. just as a print-name can be stored on a'property-list . There'is no 
reason why the alphabet shouldn't have its set of concepts, so that tfee words " 
attached to concepts vould be paade up of ordered sets of items which were 
instances of letter ^concepts. Thus letters could \e drawn on th^ screen using ^ 
t-he same general procedure for objects. Fahlman [1975] makes a similar 
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P^em that would have^ to be faced if the system'-Vere expanded is that 
of erasini .parts of objects'. There are at least two subproblems. Co;sider the 
figure produced by drawing two overlapping trian^es: 




«e-vill presume that triangle' h^s been defined for the system. If the user " 
requests ti,at parts of lines which are inside 'another triangie be erased, the 
result should^ be; • 



J- 4.- ■ ' . 

. . , . ' .. 

« , J • ' \ * 

This would involve some- geometric ^calculation, and would leave parts of de- * 

Hned parts of obj.ects do be represented. This inay be a representational 
-prqjrlem. -Another problem occurs if the user requests that tlje inside tri- 
- angle be erased: ' _ • 




The problto here is tha^ the system do\s npt "know" that the overlap of the 
two triangles is itself^ triangle.' A Is^ilar situation can occur if several • 
objects are put toiether so tHa't the spaces between them fonn some recognizable 
shape* 

The problem of object selection by touching hks been solved rather * 
^ simply injhe present system by selecting the'object if the touch coincides 
_ witho^s defined. lo^^fon point, or. aKernatively, some' other 'defined point. 
However, in-a realistic system cn6 would w^nt to allow touches to olher places 
in order to select the object. /^r example, a circle could be selected by 
yuchir^-.some point on its clrctfef ^ence, testead of its center. It should " 
also be possible to select some closed figure 'by touching some point which lies 
inside th^figure. X complete ' theory %f object selection is a hard and- ex- 
tremely interesting probl^; and would be a necessary addition to any practical 
system. _^ _ ' "■ ' ■■ ' 

' Of the probleiis presented in 'sect ion- 5, but not ' discussed here, " ' 
• the,mosx challenging are those af heuristic defaults^and the^network speci- 
fication of structural condition s^l-Whman (1977] has mS^e to say about the , ■ , 
latter problem. ' '^•>'' ■ * , ^ '■ . 

— • ™^ ' ^ ,^ - ^ ■ ^ *" , 

7.3 The Future ' ' • ' \ ' ' ' - '" 

. . This section desc^bes S^of the 'topics in which thp members of our - 

^ ^ group areVcarrying. out research. We are how using t^e NLG pilot system as ^ 
. background for r.^ ideas s^nd wi^ be co^icenprating ^'the three areas of lan- 
7\ guage analysis, knowledge representation, and protocol analysis.' 



ERIC ' ' . ^ 



Language analysis in a practical system should be robust.^' practical 
habitable system should possess the ability to respond intel^iLtly to sentences 
regardless of the completeness of the graimaar. Since..a-pw^te graanar of 
English in this practical sense does -not .exist, research in natu^ral language 
understanding must therefore develop methods that account, for all potential " . 
input. . Under investigation is the development of a language analyzer which 
. combines syntactic with semantic ^processing forming a uniform model which 
approaches this capability. It is believed that' several additional benefits 
will accrue from' this, approach. This scheme should be capable of correctly 
processing many elliptical and some^uQgralHnatical sentem;es. In- addition, 
s^ntic cohesion should override poor syntactical form. ■ This w^uld allow ' 
frequent users to invent shorthand forms conveniently with no alterations to ' 
the language processor. For e^ple. m an NLG extern one could use .(17) and 
(18) instead of longer forms.. , -\ 

(17) DRAW CIRCLE (200,300) ■ * 

(18) LINE <touch> <touch> ^ 

Work,fn knowledge engineering -will attempt to 'improve and extend the 
representation used in the pilot system, in order to provide a knowledge base 
which uses a homogeneous -represent^ion of 'many different kinds of information. 
It should, for example, include the structure of an object, how to draw it, * 
the object's function, and relationships among drawn objects. The knowledge 
.base should ^serve as a lexicon, a database of facts, and a graphical database. 
In addition, a portion of the research^ll be concerned with conv^Slon between 
the different types of information, either by spontaneoua coaputation or on , 
demand • ^ 

Protocol analysis should lead us to discover adequate vocabulary and ) , 

granE^r and preferred modes of man-i=achine interaction, particularly ratios 
of graphical to ^language use and usags patterns. Some information abou- ^ 
semantics can also be gathered by recording user reactions to systi responses. 

In addition to the effort in the topics already discussed, 'we are concerned 
vith sy^em organization, language generation; graphics, and processing drawn 
input, , ' • 
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_ t 

This report has describe a successful experimental pregrarr^or' ^pu-" 
lating Staple- pictures oh a computer - graphics terainai; Kaimra^l^Bguage and 
touch input can be used to give c<^nds to ^rav, arase and nane, and^ to' i^se 
•questions about the objects on the" screen ^ £Ee Capabilities of the systea. 

We are confident that vith' 4 robust and^powerful language analyzer, 
nixtures of language and pictures for both input and output, stored knowledge 
about the. pictures beln^ nanipulated, and an i^'erence capability to' assist 
the user, ve will have a productive research and applicationa" tool. Katural 
Language Graphics provides a fertile area for significant research and larger, 
=ore 'general hXC systems should result fron our work. 
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Worfe (Decimal) 



ERR 

EXEC 

GEN- 



GI 
GR 
.IN 
INIT 
KB 



.-LA 



Templates 

- ATN Netvork 

- LISP Code 



MENTAL Functions 
Initial Network 
LISP Code 

ATN" Network 
Lexicon- 
LrSP Code 



Misc. Functions 



- 26 
,158 

. 80 

359 
VI 99 
,23 
297 
233 
112 

1153 
3^00 
7000 

5^68 

1927^-^ 
2017 
517 



/ 
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its' 



1. 

2. 
3. 
4. 



9. 
10. 
11. 
12. 
13. 
14. 
15. 



16. 

17. 

18. 

19. 

20. 

21. 

22. 

23. 

24J 

25. 

26. 



<acn net> 
<arc set> 
<state> 
<arc> 
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<action> 
<stor*age act> 



7^ <retrieve act> 



<create act> 



<category> 

<ctgy> 

<te8t> 

<terffi act> 

<words> 

<word> 

<preact> 



<label> 

<con8tit> 

<regx 

<fonD> 

<expr-ession>' 
</roto ca8e> 
/<feature> 
<qafunct> 
<qa arg8> 
<fragment> 
<proto> 



(<arc'8et>2) 

(<8tate><arc>o) ' 
any ATN 8tate name 

(CAT <category> <teat> <action>.Q <term act>) or 
(JUMP <8tate> <test> <action>o) o£ 
(MEM <words>.<test> <action>o <term act>) or 
(POP <fonn> <test>) or — 
. (PUSH <statK<te8t>~preact>o <action>o <term act>0 or 
(TST <Label> <C^t> <action>o <term act>) or 
(VIR <con8tit> <test> <action>o <tenn act>) or 
(WRD <words> <test> <action>o <term act>) 
<stotage act> or <retrieve act> or <create act> ' • 

(Sm <ireg> <form>) or ' 
(SETRQ <reg> <expres8ion>) or 
(LfFTR <reg> <form>) or . ~ 

(LIFTRQ <reg> <expre88ion>) gt_ ' . 

(HOLD <con8tit> <form>) or . / 
^PROTOTYPE <proto^ca8e> 7form>) or ' ^ 
<forin> 1 
(GETR <reg>) or 
(GETF <feature>) or 
(RFEAT <feature> <form>) or 
(CTGY <^category>) or 
'NULLR <reg>) or 
(NEXm)RD)*or 
(PROTO <reg>) or 
(QACALL <qafunct> <qa arg8>) or 
FEATURES or — • 

* or 
<fonD> 

(BUILDQ <fraginent> <*:eg>Q) or 

(LIST <form>Q) or ^ 

(APPEND\<form> <fona>) or 

(QUOTE r^xpres8ion>) or 

(UNIONP <proto> <proto>) or 
^ <fonii> 

(<ctgy>|5) or <ctgy> 
• •= any lexical category 
: <action> or T. 
: (TO <8tate>) 

(<word>0) or <word> 

any lexical word 

(SENDR <reg> <fonn>) or "^-^ 

(SENDRQ <reg:* <expre8^on>) or 

(!.<fonn>) 
: : « a LISP atom * 
: := a LISP atom 

' • " any ATN register name ' , 

::«^'^actiQO>or any LISP expre88iotL to be EVALuated 
J 2= any LISP expre88ioD ^ 
• ^* a case aame in the prototype 
J i" atiy lexical feature 
: FINDOBJ or CONVERT 
argument list for OA 

' a LISP skeletal expre ssion using * and for subati tutionQ 
i" an NLG prototrype '* 
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A 

AN 

AND 

ARE 
AT 

BETWEEN 
• CALLr- 
r CALLED 
CAN 

CENTIMETEJl 

CENTmETERS. 

CENTIMEtRE ' 

CENTIMETRES 

CIRCLE 

CIRCLES 

CIRCUMFERENCE 

CM 

CONNECT 
CONSTRUCT ; 
COULD 
DEGREE 
DEGREES 

/diameter 

DID . 
DO 

DOES ' 

DRAW 

EIGHT 

EIGHTY 

ENDPOINT 

ENDPOINTS 

ERASE 

FIFTY 

FIRST 

FIVE 

FORTY 



FOUR 


OR 


FROM 


PLEASE 


' HALT 


POINT 


HAS 


POINTS 


HAVE 


PUT 


HERE 


RADIAN 


HORIZONTAL 


RADIANS 


HOW 


RADIUS 


HUNDRED 


SCREEN 


IKCH 


SEVtEN , 


INCHES 


SEVENTY 


IS 


SIX . 


' JOIN 


SIXTY 


KINDLY 


SMALL 


LARGE 


STOP 


LENGTH 


STRAIGHT 


LINE 


TEN 


LINES 


THAT 


LOnG 


THE 


MAKE 


THEN 


MANY 


TflERE 


ME 


TBIKG 


MILLIMETER 


THINGS 


MILLIMETERS 


THIRTY 


MILLIMEKlE 


THIS 




THREE 




THROUGH 


NAME ^ 


TO 


NAMKn 


TWENTY 




TWO 


NINE 


UNIT 


NINETY 


UNITS 


NOW 


^ VERTICAL 


OBJECT 


- WHAT 


OBJECTS 


WHERE 


OF 


WHICH 


ONE 


wm 



J 
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1 


K 
t S 


§ 
z 


Z Ui 

< 

S£ < 


<' 
?- 




-3 




o 


d 
< 




ABBREV 




i AND 


^ det" 






1 












HIl 








COKJ 
















^ AKGLE 


N 




i 


1 


i(XORIEKT) 


^ r— 












j ARE 

! Xt 


COPUL 


4 IS 


i PLUR,! 


















1 


PRE? 






t 

-r- 


• (ZORIEKT 
i XPOSIT) 


. LOCATOR j 














f r ■ ■ - — 1 t 


?RE?C 






1 


UrposiT) 


t ENDFOIKT ' i 


t 










tALL 




t 


^ -NAKE 




it. 
















* CALLED 


*CK^?L 


KAHE 




i 


(XKAKE) 




1 — 












CA^* 




ALT. 




























CIRCE 




N 








! (ZOEJECT) 


-r-^^- 


















CK 




K 




?LUR 


i 


iCZSIZE) 










LIKE 






T 




CONTCECT 




Y 


DRAW 












-LIKE 














DEGREES 




K 


- DEGREE 


- ?LL\Ri 


. (ZORIEKT 


1 








1 AKGLE 










DRAV 




V 






fr 




T ^ 

J 


1 




1 

1 




f 

1 . 








< - EIGHT 




NTH 












\ 




i 


! 








• ERASE 




' V 






il 














i — p 




FIRST 




0?J) 










* 


! 










i 




FROH 


f 


P?>E? 




4 




f(Z?OSIT) 


tEKDPOINTi 




• — ^ 1 


1 








HERE 




ADV 


?— 






(%?OSIT) 


\ 


\ 
















■ ROW 


T 


OADV 






— « — 


•5 

s 


















INCH . 












'(XSI2E; 










LIKE 










1 UL^^lCE 






256 






I iXSlZZ) 


i 














LIKE 
'^-LIKE 


i 
i 


}i 

K 


ST-LI?.a 
Cl*R\T 


i 

1— 


K'ZOEJECT) 
1 (ZSSJECT) 


L ^ 1 

i 






(STRAIGHT) 








^-is 




PROK 
























or 


*?RE?'^ 

f 




1 




|(ZSI2E 
. ZORIEKT) 
















: ^ PLEASE 




ADV 








1 


\ 






4 








» n)iNT ■ .\ f - i 




CZ03JECT) i 


f— 

f 














RADIUS 








i(ZSl2E) i 




(CIRCLE) 












[ SCREEK 


1 N < 






(ZOBJECT) 


















SHALL 




ADJ ^ 








(XSUE) 




\ 














THAT 
THROUGH 


DE?i f 
PRE? i 








(tPOSIT) 
(ZPOSIT) 


LOCATOR 
















VERTICAL 


AIXJ 1 


90 






(XORI^T)I 


















WHAT 
WHERE 


QWORD j 








(ZOBJECT 
ZK^) 


















VHICti 


o&eTj 


























WITH 


Pkrr ^ 

! 








(ZSUE 
ZORIEOT 
XPOSIT) 


EKDPOINT 






















4 
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Appendix E; Sentence-Level Granmar 
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Appendix J: Sample Sentences and Their Parse Times 53 
SENTENCES ^ 



TIME (m^J / 




» 

Draw a straight line from hei^ <T> to there <T> 
. Please make froci here <T> a Une to there <?> 
Please make me a line from (234, 412) to here <T> 
Please erase the line Here' <T> [ - 

Draw a line between here <T> and here <T> - 
Put a point here <T> 

Draw a point named FRED at <T> ^ 

Draw the point ETHEL here <T> * 

Now connect FRED and ETHEL 

Draw, a straight line from <T> through <T> 

Join the point FRH) with this point <T> * 

gtop 

Please connect this <T> with this <T> ' ' " 

From here <T> draw a lin? to there <T> 

FrcJa this point <T>. draw a line to FRZD 

<T> <T> Draw a straight line fro^a here to here 

With this <T> connect this <T> 

'Draw a three inc'rl ling named L3 here ^> 

Draw a nn^isontal line with a 3 inch length here •<T> 

Draw a vefCtical line 5 inches long called L5 to here <T> 

Draw a circle with a 2 inch radius here <T> \ 

Please draw a hundred ^ line named DAVE at' an aZle of 
fifty degrees here'<T> / 

Make me a circle called HEAD with a 2 inch diameter there-<T> 

Erase the line named ARM 

Eras^ the line from here <T> to here <T> 

Draw a circle of radius four units named EYE here <T> 

Eras^^this circle <T> . 

Call thi^<T> X43 » ' * , ^ 

Name the three inch line L3 
What can you draw? 

Can you draw, circles? ^ 
Can you draw a circle with a 3 inch circjRrence? 

How many screen units i^n inch? \ ^ 

Is there a point named P? ' ^ 
What did you draw at P? ' ' 

Can you draw? ., . 59 ' . 




2J3 
366 
366 

150 

poo 

116 
217 
200 
167 
317 
266 
17 
266 
333 
^33 
•917 
333 
217" 
350 
317 
300 

401 " 

266 

167 

316 

250. 

133 

150 

216 • 

67 
100 
317 
250 
133 
183' 
100 
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<iaessage> 
<prototype> 



<type> 

<act> 

<obj> 

<name> 

<ornt> 

<posit> 

<pc> ^ 

<coords5 



<x-val> 
<y"val> 

<node identifier> 
<si2e> 



<nuEi> 



(KB LA #Q (MG <prototype>)) 
( (ZTYPE . <type> ) 



(2ACTI0K 
(%OBJECT 
(ZNAME 

(%ORIEHT 

(2P0SIT 

(2SIZE 



o <act> ) 
^ <obj> ) 
<name> ) 
<ornt> ) 
<posit>) 
, <si2e> ) ) 



IMP or Q 



: :« DRAW or ERASE or NAME 

ST-AIKE^or CIRCLE or POIKT 

: := a LISP atoa * 

' ' a number in degrees 

:r» (<pc>t) 

(EKDPOINT <coords>) or 
(UXATOR <cx)ords>) or 
(TOUCH <coords>) 



(<x-val> 
(NODE 



<y-vai>) or 

<node identifier>) 



x-^coordiaace in screen units ' 
y^coordiante in screen units 

the identifier of a node in the seaantic netvork 



(RADIOS 
(DIAMETER 
(CIRCUMFEREHCE 
<nuiii> 



<nui!i>) or 
<nuzs>) or 
<nxm>) or 



a' length in screen units 



\ 
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TEMPLATE 
NUMBER 



.TEMPLATE 



TO • 0 

. Tl ^ (OK) 

^■2 (Thanks *.for the session) 

"IJ^' ^ ' (Satural Langtiage Graphics systen)- 

IZ ' - -Pilot Systea - sucEser, 1976) 

(Please rephrase the sentence) 

(Object as specified cannot be found) 
T7 (*) 

T8 (yes) 
T9 (no) . 

■TiO (I don't knov) 

"Tii (*) 
T12 . (* and *) 

■^13 - , (* , * , and *) — 

T20 ^ - (at *) 

^21 • (* has no name) 

(a * nased *) 
^23 (a *) 
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• A 



NAME 



PART OF 



ACT-1 
ACT-2 
ACT-3 
ACT-4 
ACT-5 
ACT-6 
kCt-7 
ACT-8 
.ACT-9 
ACT-10 
ACT- 11 
ACi;-12 
ACT- 13 
ACT- 14 
ACT- 15 
ACT- 16 
ACT- 17 
•ACT-18 
ACT- 19 
ACT-20 
ACT- 21 
ACT- 2 2 



POINT 
■ST^LINE 



ST-LINE 
ST-LINE 



POINT 
DEFPOINT 

DISTANCE 
A2IGLE 

CIRCLE 
CIRCLE 
CIRCLE 
CIRCLE 



; 



J 



FUNCTION 



How to draw a POINT 
Structural check 



How Co find midpoint 
How to draw a ST-LINE 



J 




. — ' L 

■Strutt-urai check 
Structural check ^ ' • 

Structural check 
Structural check 

Structural checl^ 
How to draw a CIRCLE 
How to find a locator 
How to find a radius 
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Appendix 0 : KBNET - Initial State of KB Network 

( (DEFINE 

SUBSETOF SUBSETOF- 
_ELMNTOF ELMNTOF- 
NAME NAME-' 

TYPE TYPE- • J. . * 

ELMinDESCR ELMNTDESCR- 
DATTR DATTR- 
ROLE ROLE- 
RESTR RESTR- 
ACTION ACTION- 
ATTR ATTR- 
INSTNCOF INSTNCOF-- 
VAL VAL- 
CASE CASE- 
r NUMBER NLTCBER- 

^OBJECT OBJECT- " ♦ 

^SED- 

JtRttTURAL STRUCTURAL- 
I-NSTANTIATES INSTA>mATES- - ^ ' 

)) ' - . 

((BUILD TYPE SYSTEM NAME SCREEKMODEL) ) ' - 

({BUILD NAME OBJECTS TYPE SET)= XX) ' 
((BUILD }iAMZ LINES Vi?Z SET SUBSETOF *XX)= XY) 
{(BUILD NAME PRIMITIVES TYPE SET SUBSETOF *XX)- XZ) 
((BUILD NAME DEFPOINTS TYPE SET SUBSETOF *XX)) 
((BUILD NAME ANGLES Ti?Z SET SUBSETOF *XX)) 
((BUILD NAME DISTAltTES TYPE SET SUBSETOF -^XX) ) 
((BUILD aHVES TYPE SET SUBSETOF *XY)) 

^(/BUILD ST-LINES TYPE SET SUBSETOF *XY SUBSETOF *X2)) 

((BUILD NAME POINTS HTE SET SUBSETOF *X2 SUBSETOF (FIND NAME DEFPOlkft) 

('(BUILD NAME UNKERSE TYPE SET - . - _ 

SUBSETOF- • *XX 

SUBSETOF- (BUILD NAME SITUATIONS^ 
TYPE SET 

SUBSETOF- (BUILD NAME CX)NNECTIONS TYPE SET) 

^ " ) 

)) - • > I 

((BUILD f^AME DEFPOINT . ' - ' ' < 

TYPE CONCEPT ^ ^ - 

ELMNTDESCR- (FIND NAME DEFPOINTS) ' i 

STRUCTURAL ACT-14 ^ 
f)ATTR (BUILD ROLE XVALUE - 
TYPE DESCRIPTION 
^ " RESTR (FIND 'name DISTANCES) 

DATTR (BtJILD ROLE YVALUE I ' 



TYPE DESCRIPTION f 



) 
)) 

({BUILD NAME DISTAI^^TE 
TYPE* CONCEPT 

ELMNTDESCR- '(FIND NAME DISTANCES) 
STRUCTURAL ACT-16 



RESTR (FIND NAME DISI^CES) . ^ 



)) 



((BUILD NAME ANGLE" 

TYPE gOKCEPT ' 
ELMNTDESCR- (FIND NA>IE ANGLES) 
STRUCTURAL ACT-17 

)) 

((BUILD NAME-POINT ' 
TYPE CQHCEPT 

ELMJrrO.ESCR- (FIND NAME'tOINTS) • 

STMJCTURAL ACT-13 

DATTR (BUILD' ROLE LOCATOR 

- TYPE DESCRIPTION* 
CASE ZPOSIT 
^ RESTR (FIND NAME DEFPOINTS) 

DATTR (BUILD ROLE TODRAW 
' - — TYPE DESCRIPTION 

ACTION ACT-1 
CASE %HOH 

) 

)) 

((BUILD NAME ST-LINE 

-- TYPE CONCEPT. * 

ELMNTDESCR- (FIND NAME Sf-LINES) 
STRUCTURAL ACT-2 
iATTR (BUILD ROLE ENDPOINT ■ 

TYPE DESCRIPTt&J{ 
^ RESIR (FIND 'NAME^ DEi3»0INTS) 

- DATTR (BUILD ROLE ENDPOINT 

. TYPE DESCRIPTION 
_^ RESTR (FIND NAME DEFPOINTS) 

DATTR (BUILD ROLE LOCATOR 

TYPE DESCRIPTION 
CASE 2P0SIT 

» RESTR (FIND NAME DErPOIOTS) 

) ' 
DATTR (3UILD ROLE MIDPOINT 

TYPE DESCRIPTION 
RESTR (FIND NAME DEFPOINTS) 
_ ACTION ACT-5 

DATTR (BUILD ROLE ORIEKTmON 
TYPE DE^CRmiON 
CASE %ORIENT 
^ RESTR (FIND NAME ANGLES) 

DATTR (BUILD ROLE LENGTH •, 

.TYPE DESCRIPTION 
CASE %SIZE 
^ RESTR (FIND NAME DISTANCES) 

DATTR (BUILD ROLE TODRAW 

TYPE DESCRIPTION 
CASE 2H0W 
ACTION ACT-8 

■) 



)> 
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((BUILD NAME CIRCtES 
TYPE SET 
- SU8SET0P (FIND NAME CURVES)- 
' Eli^NTDESCR- (BUILD NAME CIRCLE 
'• • TYPE CONCEPT 



^ ) 

)) 



. STRUCIJJRAL ACT-19 ' ' ■ 

DATTR (BUILD ROLE LOCATOR 

•fYPE DESQIIPTION 
ACTION ACT-21 ^ 
CASE 2P0SIT' 
■ , ^ I^STR {FIND NAME DEFPOINTS) 

DATTR (HJILD ROLE RADIUS . 
TYPE DESCRIPTION 
ACTION ACT-22 

CASE ZSIZE ' 
^ RESTR (FIND "NAME DISTANCES) 

DATTR (BUILD ROLE CIRCUMFERENCE 
TYPE DESCRIPTION ' 
' RESTR (FIND lUm DISTANCES) 

DATTR (BUILD ROLE PIAMETER 

TYPE DESCRII^ION 
^ ' RESTR (FIND NAME DISTANCES) 

DATTR (BUILD ROLE TODRAW 

TYPE DESCRIPTION 
CASE ZHOW 
ACTION ACT-20 

) 



STOP •(*** TO FORCE WAY OUT OF MENTAi READ LOOP ***) 



\ 

Note: The im^rs^of the links are indicated by the addition of a dash. 
Note: The above description is in the form read by the MENTAL interpreter. 




Appendix P ; The GR Module Interface 

The interface, at GR^^ defined as follows: 

<message> iGRKBjQi <protbtype> )) 

<prototype> (• ^ZTYPE . sooe type ) 

. (ZACTIOH . some action ) 

' ' (ZOBJECT ; some object ) 

X%NAM^ , some name ) 

^ • (%HOW . <progrant> ) ) 

<prograin> (<step>j^) ^ ^ 

<step> . '(FAPKTxl^D 

or 

(F4LIKE1 xl 21 2£2 jsr2) 

or 

(F4LIKE2 jcl yl angle length) 

or 

(F4LINE3 midx mldy x2 ^2) , 

or 

(F4LIKEA midx midy angle length ) 

or 

(F4CRCL cx £2 radius ) 

or 

(F4^SE) 
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Appendix Q : 

The Syntax of Calls to the QA Sectionof the k1 Modu le 

<:iDessage>^ ::= ' (KB LA #q' ( QA <functlon> » 
<function> (INFO <keywords> <prototype form>) 



or 



or 



or 



^ 4 

(FINDOBJ <name> <rtype>) 
'(OONVERT a nmnbet^ <units>) 
'(FINDDRAW <object> ) 



<keywords> < := (<key5>j^ ) 



%OBJECT or %HAME or \pOSIT or NODE 
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<PK^ <key5> \\' 

<prototype form> any valid complete or partial prototype 

^ an object nAme extracted from an ioput sejitence 
: :^ ST-LINE or POINT or CIRCLE ot m\r^ 
: : = mil or INCH or CM or MM or DEGREE or RADIAN 
•= a primitive or suspect^ object 



<name> 

<type> 
r 

<UDitS> 

<object> 
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